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Abstract 

We first show how a new definition of entropy, which is intuitively very 
simple, as a divergence in cluster-size space, leads to a generalized form 
that is nonextensive for correlated units, but coincides exactly with the 
conventional one for completely independent units. We comment on the 
relevance of such an approach for variable-size microsystems such as in a 
liquid. We then indicate how the entanglement and purity of a two-unit 
compound state can depend on their entanglement with the environment. 
We consider entropies of Tsallis, which is used in many different real- 
life contexts, and also our new generalization, which takes into account 
correlated clustering in a more transparent way, and is just as amenable 
mathematically as that of Tsallis, and show how both purity and entangle- 
ment can appear naturally together in a measure of mutual information 
in such a generalized picture of the entropy, with values differing from 
the Shannon type of entropy. This opens up the possibility of using such 
an entropy in a quantum context for relevant systems, where interactions 
between microsystems makes clustering and correlations a non-ignorable 
characteristic. 

1 Introduction 

A new form of generalized nonextensive entropy recently proposed by us[llj 
has been shown in to give small but interesting departures from the Shannon 
case in terms of thermodynamic properties of a system in a manner similar to 
but also somewhat different from Tsallis entropy [131 21 [TU] . Conceptually, the 
new entropy appears from a novel definition of entropy in terms of the rescaled 
phase cells due to correlated clusters, and in a limit similar to the Tsallis case 
approaches Shannon's classical extensive entropy. Kaniadakis has also suggested 
[6l [7] the use of deformed functions leading to unusual forms of entropy, from a 
kinetic Principle related to phase space, which gives excellent results for cosmic 
ray spectra. In our case the definition of entropy is particularly simple, as it 



1 



can be expressed simply as the divergence of a vector representing the modified 
probabilities for the different possible states taking into account a rescaling due 
to correlations or clustering due to interactions between the microsystems. 

In microscopic systems quantum entanglement of states is also a relevant is- 
sue. Some authors [T51 III HI have studied the problem of quantum entanglement 
of two states in the picture of Tsallis type nonextensive entropy. The generaliza- 
tion of Shannon entropy to the very similar von Neumann entropy using density 
operators in place of probability distributions [5] reveals common features of the 
stochastic and the quantum forms of uncertainties and this treatment can be 
extended to Tsallis' form too. 

Our purpose here is to present a combined study of stochasticity and quan- 
tum entanglement, so that the former emerges from the quantum picture in a 
natural way, and then we intend to show that our new approach of defining 
entropy also allows us to obtain a measure of mutual information that involves 
stochasticity and entanglement together in a clear comprehensible way. The 
fact that our new definition of entropy, which is conceptually very simple, also 
gives the probability distribution function in a closed form in terms of Lambert 
W functions [11] allows one to carry out many calculations with the same ease 
as for Tsallis entropy. In this work, however, the probability distribution will 
not be needed for explicit use. 

2 A Generalized Form of Entropy from a New 
Definition 

Entropy is intuitively associated with randomness, because it is a measure 
of the loss of information about a system, or the indeterminacy of its exact 
state, which in turn depends on the probability distribution for various states. 
A uniform probability distribution function (pdf) among all states indicates 
maximal uncertainty in state space and gives the maximal entropy, whereas 
Dirac/Kronecker delta (continuous or discrete states) pdf with no uncertainty 
has zero entropy. Combinatorics gives the Boltzmann form 



because in equilibrium the nj are simply Npi, where N is the total number 
of subsystems, and n% is the number of subsystems in the i-th state. In terms of 
the pi themselves one gets the Shannon form given below. It is well-known that 
maximizing the entropy with the constraints ^2 Pi — 1 and (with Ei the energy 
of the i-th state, and U the total energy, which is fixed) J^Pi^i = U gives the 
exponential probability distribution 



where the Lagrange multiplier constant [3 can be identified as the inverse of 
the temperature. 



S = fclog[J\n/JJni!] 



(1) 



Pi = Cexp(-0Ei) 



(2) 
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Let us now consider Shannon coding theorem [S] : when the letters Ai of the 
alphabet used in a code have the probabilities pi, then it can be shown fairly 
easily that a stream of random letters coming out of the source with the given 
probabilities in the long run will relate the entropy to the probability of the 
given sequence: 

P(sequence) — JJpf*'* = exp[— NS] (3) 

i 

where S is the entropy per unit and N is the large number of letters in the 
sequence. 

We shall now define entropy from a somewhat different viewpoint which 
takes into account interaction among the units, producing clusters of size qi 
units for the i-th state. This effective size may in general be a fraction, and 
if the interaction is weak, the average cluster size qi is just over unity. If we 
think of liquid clusters, the typical subsystem in state i may bean assembly of r» 
molecules, but this may change due to environmental factors, such as pH value, 
to Si, so that we have a rescaling value of qi — st/ri, which may be greater or 
less than 1. In general we allow qi to be different for each i. Since pi is the 
probability of a single occurrence of the i-th state, i.e. for a cluster of size unity 
(which may consist of a typical number of subunits), the probability for the 
formation of a cluster of size qi is p(qi) = {pi) qi ■ Let us now consider the vector 

= (4) 

This is rt-dimensional, where n is the number of single unit states available 
to the system components. 

Let us now consider the "phase space" defined by the % co-ordinates. As we 
have said above, the deviations of these parameters from unity give the effective 
(which may be fractional when an average is taken) cluster sizes in each of the 
states. A value smaller than unity indicates a degeneration of the micro-system 
to a smaller one in a hierarchical fashion, partially if it is a fraction. In other 
words we are considering a scenario where clusters may form superclusters or 
be composed of subclusters, with a corresponding change of scale in terms of 
the most basic unit obtainable. We have dealt elsewhere with the interesting 
question an oligo-parametric hierarchical structure of complex systems |12) , but 
here, we restrict ourselves to cluster hierarchy changes that do no qualitatively 
change the description of the system. 

Hence, if we take the divergence of the vector m in the qi space, it is a measure 
of the escape of systems from a given configuration of correlated clusterings. 
And, inversely, the negative of the divergence shows the net influx of systems 
into an infinitesimal cell with cluster sizes qi. If all the qi are unity, then we have 
unfragmented and also non-clustered, i.e. uncorrelated units at that hierarchy 
level. 

We can argue first from the point of view of statistical mechanics that this 
negative divergence or influx of probability, may be interpreted as entropy. 
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(5) 



-^log(pi)pf 



We know that the free energy is defined by 



A = U — TS 



(6) 



where A is the free energy, U is the internal energy, and S is the entropy. 
T is the temperature, or a measure of the average random thermal energy per 
unit with Boltzmann constant k chosen to be unity, and hence ST is a measure 
of the random influx of energy into the system due to the breaking/making of 
correlated clusters due to random interactions in a large system. This allows the 
subtracted quantity A to be the free energy or the "useful" energy. There are 
usual thermodynamic phase space factors in dealing with a macroscopic system, 
which we drop as common factors in what follows. 

In terms of the Shannon coding theorem also we arrive at the same expres- 
sion. Since, with qi average clustering the i-th state occurs with probability p(' , 
a stream of units (clustered) emitted will correspond to the probability 



which too gives us Eqn. [5) 

In [llj we have developed the statistical, mechanics of this entropy in detail, 
by first obtaining its probability distribution function in terms of the Lambert 
W function. However, we used, for simplicity an isotropic (in state space) 
correlation and rescaling, i.e. we had a single common q, as in Tsallis' entropy. 
In the rest of this work we shall use the same simpler expression. 

3 Quantum Entanglement and Stochasticity 

When the state vector in the combined Hilbert space H of two particles (or 
subsystems in Ha and Hb) cannot be expressed as the factorizable product of 
vectors in the Hilbert spaces of the two subsystems, it is by definition entangled. 
Hence entanglement is actually a property related to projection in the subspaces, 
and cannot be expected to be measurable by properties in the bigger space alone. 
Given the state 



exp[-S] = Y[pf 



(7) 




(8) 



We get the density matrix for the product space 
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PAB = CijCki\ij)AB{kl\AB (9) 

which gives density matrix for the Ha space for the trace over the H B part 



Pa = CC^ (10) 

where the C's are now the coefficient matrices. In terms of density matri- 
ces an entangled state (for an explicit example) of two qubits ( a "qubit", or 
quantum bit, being a quantum superposition of two possible states) may be 
expressed by the reduced 2X2 matrix from the |0)|0), |1), |1) basis sub-set: 



P 



cr 
7cs 



7cs 
■2 



(11) 



with 7 = 1 for the pure quantum (entangled) state 



|V)=c|0>|0)+*|l)|l) (12) 

and we have used the compact notation c = cos(0), and s = sin(0). This 
entanglement occurs in the subspace of the product Hilbert space involving only 
the two basis vectors 1 00} and |11). Other entangled combinations are equivalent 
to this form and may be obtained from it simply by relabeling the basis vectors, 
and hence we shall use this as the prototype. 

For I7I < 1, we have an impure state with a classical stochastic component 
in the probability distribution, although we still have probability conservation 
as Tr(p) = 1 which remains unchanged under any unitary transformation. A 
possible measure of the factorizability ( "purity" [14] ) £ of a quantum state, or 
its quantum non-entanglement, remains invariant under changes of 7 



( = Tr A [Tr B (p AB ) 2 } 

= c 4 + s 4 . (13) 

So, for the maximum entanglement £ = 1/2 when 9 = 7r/4, and the minimal 
entanglement corresponds to £ = 1 (pure factorizable states) when 6 = 0, ir/2. 

Quantum impurity represented by classical stochasticity attains the maxi- 
mum value when 7 = 0, and is nonexistent when 7 = 1 corresponding to a pure 
entangled state. 

We note that £ does not involve the stochasticity-related parameter 7 at all, 
but remains the quantifier of the quantum entanglement. 

Another equivalent but conceptually possibly more interesting way of quan- 
tifying entanglement may be the parameter 



E AB = 2{Tr[ PAB ] - Tr[p A ]Tr[p B ]) = sm'(20) (14) 

which is more symmetric in the two subspaces and is similar to a correlation 
function. It gives for no entanglement when 8 = 0,7r/2, and maximal entan- 
glement 1 for 9 = 7r/4, as desired. This definition of entanglement is in the 
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spirit of mutual information, though wc have not used the entropy at this stage, 
but only the probabilities directly. It too docs not involve the stochasticity in 
terms of the purity parameter 7. In the relation above we have used 



p A = Tt b [pab] 
and similarly for pB- In our specific case, for A or for B, 

c 2 



PA,B 







(15) 



(16) 



with Tr[pA,B\ = 1 ensured. 



4 Stochasticity from Entanglement with Envi- 
ronment 

It is possible to formulate the stochasticity by coupling the entangled state 
Y&ab > to the environment state \^e > quantum mechanically and then taking 
the trace over the environment states. 

\^ABE) = Y^Cijk^A^B^E (17) 

ijk 

Hence the density matrix for the pure quantum system is 

PABE= c i3kC^ mn \ijk)(lmn\ (18) 

ij k .Ivan 

and the trace over the environment yields 

Pab = c ijkCi mk \ij)(lm\ (19) 

ij,lm 

For the entangled mixture of 1 00) and |11) in Hab and the couplings 



cooo = cc 
cooi = cs 1 
cno = ss' 
cm = sc 

with c' = cos(0') and s' = sm(8'), 

the trace over He states gives the density matrix 



PAB 



2csc's' 



2c's'cs 



(20) 



(21) 



So, classical stochasticity has been introduced by taking the trace over the 
environment space with 



G 



7 = sin(20') 



(22) 



5 Entanglement, Entropy and Mutual Informa- 
tion 

5.1 Single System Interacting with Environment 

Let us consider a single system A interacting with the environment E. The 
product space Ha ® He contains entanglement between the measured system 
and the environment, and hence the density operator for the combined system- 
environmental space is as given in Eqn. [IT] with 7 — 1 to indicate a pure 
entangled state, and the environment-traced density is given by Eqn. 1151 Here 
Pa and pe are equal. Mutual information may be identified as the entanglement, 
and defined by 

E AE = 2{Tr AE [pA E ] - {Tr A [p A }) 2 ) = sin 2 (20') (23) 

as before, with 0' the angle of entanglement. Hence, measurements on the 
system A reflects the coupling of the system to the environment to the environ- 
ment, and the mutual information is contained in the parameters of the system 
itself. In terms of von Neumann entropy, which, with mixing in an orthogonal 
quantum basis, becomes similar to Shannon entropy [S], 

S =-Tr [plog(p)} (24) 

We know from the Araki-Lieb relation[3] 

Sae>\S a -S e \ (25) 

that, with Sae = in a pure quantum state, we must have Sa — Se , and 
hence 



Iae = Sa + Se — Sae = 25a 

= -2Tr A [p A \og(p A )} (26) 

which too confirms the view that the system itself contains in its parameters 
the mutual information in such a case, as we found above. 

If we use our new form of entropy [TT] with the hypothesis that the mutual 
information is still given by the same form with the parameter q not equal to 1, 
which is the case for Shannon entropy, then we get 

Iae = -2Tr A [p 9 A log{p A )] 
= -2c' 29 log(c ,2 «) - 2s' 2q log(s ,2<? ) (27) 
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5.2 Entangled Systems interacting with the Environment 



With the 3-system entanglement shown in Eqn. [T7] and the relatively simple 
choice of couplings in Eqn. 1201 we have already shown pab in Eqn. 1211 Sim- 
ilar construction of Pae, Pbe and pabe, an d defining the 3-system mutual 
information as 



I abe {q) = Sabe (<z) + Sab (<?) + Sbe (q) + SAE{q) 

~Sa(q) - Sb(q) - Ssiq) 



(28) 



and with Sabe(i) = for any q, for a single 3-system pure state, we may 
find the 3-system mutual information. 

Tracing over the B space gives pae, which, using as basis |00), |01), 1 10} 
and 1 11) in the \AE) product space, yields 



c 2 c' 2 


c 2 cV 








c 2 cV 


cV 2 














S V 2 


s 2 c's 








s 2 cV 


s 2 c' 2 



PAE 



and an identical matrix for pbe- 

Finally we get using the relevant eigenvalues 



(29) 



lABE(q) = c' 2 « log(c' 2 «) + S > 2 « log( S ' 2 «) 

-\\\og{\\) - \1\og{\<L) (30) 

where the eigenvalues Ai and A2 are for the pab matrix obtained after tracing 
over E'-space. 

A + ._ = (1/2)(1±V[1-4(1- 7 2 ) C 2 S 2 ]) (31) 
with 7 given by Eqn. 1221 

Had we started with a stochastic picture of entangled impure A-B system, 
with 1 — 7 representing the stochasticity, as we have suggested above, then the 
mutual information would be 



Iab {q) = ~SAB(q) + SA(q) + 5"s(g) 
= A^log(A^) + AMog(A 9 _) 
-2c 2 « log(c 29 ) - 2s 2q \og(s 2q ) (32) 

In Fig. [1] we first show the mutual information (MI) calculated according 
to the Shannon form of the entropy, which is equivalent to our form at q = 1 , 
as a function of the A — B entanglement angle 9 and the entanglement angle of 
{A — _B)-system with the environment 9', which is related to the stochasticity 7 
as explained above. We note that mutual information is virtually independent 
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of the angle of entanglement with the environment 9'. Hence, it seems that 
traditional entropy in this case is insensitive to details of coupling with the 
environment when the mutual information between two systems is measured. 




Figure 1: Iab as a function of entanglement angle 9 in A-B space and the en- 
tanglement angle 9' with the environment, which is related to the stochasticity. 

In Fig. [2] and in Fig. [3] we show the deviations AIab of our MI from the 
Shannon MI, as a function q and entanglement angles 9' and 9 respectively, 
keeping the other angle at 7r/4 in each case. There is symmetry around tt/4. 
The variation is fairly smooth for fixed 9', i.e. fixed entanglement with the 
environment. However, if the entanglement between A and B is kept fixed 
at near 7r/4, then the mutual information using our form of entropy changes 
sharply with 9' near the symmetry value 9' — 7r/4. One can see that this comes 
from one of the eigenvalues of the density matrix pab approaching zero for this 
mixing value, and with q ^ 1, the there is either a sharp peak or dip compared 
to the Shannon entropy case, which has fixed q = 1. 

Fig. [2] shows little variation with changing 9' for almost any q. Fig. [3] shows 
pronounced changes at small q ( q < l)for different 9. 

In Fig. |4]and Fig. [5]we show the difference between our MI and the Shannon 
MI as a function of 9 and 9' simultaneously, keeping q fixed at 0.8 and at 1.2. 
Of course at q = 1, we get no difference, as our entropy then coincides with 
the Shannon form. Here too we notice that the mixing angle between A and 
B shows fairly smooth variation, but 9' or equivalently the stochasticity, causes 
pronounced peak ( for q < 1), or dip (for q > 1). Here too we can conclude that 
our method of entropy calculation can indicate a greater role of the entanglement 
with the environment when this mixing is nearly equal for the A — B entangled 
states. 

In view of the prevalent familiarity with the Tsallis form of nonextensive 
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Figure 2: Difference of MI from our entropy with that from Shannon entropy 
at 6 = 7r/4. 




Figure 3: Same as Fig. [2] with 6' = tt/4 



entropy, we have previously [11] compared our form with results from Tsallis 
entropy in formulating a general thermodynamics. There we showed that despite 
the conceptual and functional differences between Tsallis entropy and our new 
form, the results are very similar if we take the Tsallis q to be twice as far from 
unity (the Shannon equivalent value) as our value of q. In Fig. [6] and Fig. [7] we 
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1.5 



Figure 4: MI difference between our entropy form and Shannon for q = 0.7 




Figure 5: Same as Fig. [4] but for q — 1.3 



show the difference of our MI from that derived from Tsallis's entropy. We again 
note that among 6 and 9' differences are both relatively more significant for q 
values different from 1, for both angles near pi/4, with peaks and dips similar to 
the comparison with the mutual information calculated with Shannon entropy. 

It is interesting to note here that recently in a study of the entropy of a 
chain of spins in a magnetic field [5] it has been found that both the usual 



11 



1.2 



Figure 6: Difference between MI from our entropy and Tsallis's with = n/4 




Figure 7: Same as Fig. [6] but with 6' = 7r/4. 



von Neumann and Renyi forms of entropy yield nonzero and surprisingly simple 
closed expressions. Though this work does not mention entanglement explicitly, 
the correlation functions presented here, which determine the density matrix 
and hence its diagonalized form needed for entropy calculation, actually are 
manifestations of the entanglement among the spins and between the spins and 
the magnetic field. The chain has been split into two parts, similar to our A and 



12 



B subsystems, and the external magnetic field acts like the environment we have 
introduced in this work. Though they carry out their extensive calculations at 
zero temperature, unlike our finite temperature treatment, the fact they obtain 
nonzero Sa for the first L spins is apparently due to the segmentation of the 
pure state of the fully entangled quantum system and the consideration of part 
A only for the entropy calculation, which effectively is equivalent to summing 
the states of part B and the entanglement with the environment, and produces 
entropy due to the corresponding loss of information about state of the whole 
system. Hence, their results for this explicit model is consistent with our general 
result that classical stochasticity and entropy may be a reflection of segmented 
consideration of bigger complete systems. The values of the entropy of different 
types such as the canonical Shannon form or generalized forms such as the 
Renyi form, which goes to the Shannon form in the usual limit, like that of 
the related parameter we have mentioned for Tsallis entropy and for our new 
form of entropy in this work, reflect the extent of entanglement or interaction 
or, equivalently, correlation. In their work a length scale comes out of this 
segmentation, which appears to be similar to the angle of entanglement in our 
case. We do not get a phase transition as they do, because we have considered 
a simplified general finite system of only two or three component subsystems, 
not an infinite chain, and finite systems cannot show any phase transitions. 

6 Conclusions 

We have shown how a simple definition of the entropy in terms of influx of states 
into cells of any given cluster sizes in various states can give us a new nonexten- 
sive form of entropy with a closed form of the probability distribution function, 
and which coincides with the usual form for uncorrelated microsystems. We 
have then seen that classical stochasticity can be derived from quantum entan- 
glement with the environment, and it influences the mutual information between 
quantum states. The functional form of the MI differs as per the definition of 
the entropy, but the numerical differences in MI resulting from various forms of 
entropy usually differ rather subtly according the parameters of entanglement, 
within the system, and with the environment, as well as the scaling type pa- 
rameter q introduced in the Tsallis form and in the new form introduced by 
us. However, for the angle of entanglement with the environment our entropy 
differs from both the Shannon entropy and Tsallis entropy when both angles of 
entanglement are near the symmetry point 7r/4. The differences between the 
forms become more pronounced as q varies from unity. Entanglement and mu- 
tual information are such fundamental concepts that experimental tests need 
to be designed to distinguish from minute quantitative differences the appro- 
priateness of various theoretical forms of entropy. Theoretical works like the 
study of large systems such as spin chains [5] may also help differentiate the 
appropriateness of various forms of entropy such as Shannon, Tsallis, Renyi or 
our suggested new form, and the quantification of mutual information. The 
notion of clusters changing constantly into various sizes, which is the basis of 
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our definition of the new form of generalized entropy may be the most relevant 
concept for the treatment of liquids and other material, where such phenomena 
form an integral part of the dynamics. 
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